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Abstract 



This study focused on the relationship between instructional strategies and 
students' reported meta-skill attainment in an adult education context. Using the data 
collected from 2,567 end-of-course surveys of students in the Graduate Studies in 
Education program at Indiana Wesleyan University, analyses were run testing for 
correlations between the independent variables (19 instructional strategies) and each of 
ten dependent variables (10 meta-skills). 

The three key findings of this study were the following: 

> Instructional strategies clustered into four identifiable composites, each of which was 
supported by a current and credible educational theory; 

> The curriculum composite of instructional strategies strongly correlated with reported 
attainment of all ten meta-skills; 

> The assessment composite of instructional strategies correlated with reported 
attainment of no meta-skills. 

Implications were drawn from this study were the following: 

> Curriculum and assessment are essential components of any educational program and 
require substantial resource investment and attention to design. 

> Direct meta-skill instruction should be a component of courses offered in formal adult 
education programs. 
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Introduction 



The 21 st century is upon us. The Age of Information dominates our national 
culture and thought. In response to rapid economic and cultural change, adults are 
entering institutions of higher learning in increasing numbers. This shift in educational 
demographics has forced a shift in educational programming. Colleges and universities 
struggle to develop educational opportunities for this non-traditional student population. 

With a commitment to serving non-traditional students comes the need to 
understand the key characteristics of this adult learner population. The adult learner 
brings to the classroom a wealth of experience. The educator of adults must draw out and 
build upon this experience (Houle, 1996). The adult learner has a strong desire for 
relevancy. The educator must show how new knowledge and skills relate to everyday 
situations (Stein, 1998). The adult learner requires an accessible, convenient educational 
opportunity. Semi-independent educational structures for adult learning must be 
designed to meet this need (Knowles, 1980). The adult learner faces rapid and unsettling 
change in the workplace, and needs the skills critical for success in the unpredictable 
work environment of the 21 st century (DeJoy, 1997). 

The Adult and Professional Studies (APS) Division of Indiana Wesleyan 
University is an educational model designed to meet the personal and professional needs 
of the adult learner. As a result, its curriculum is constructed to be current, relevant, and 
applicable to students' personal and professional lives. APS offers its constituency nine 
undergraduate and graduate degree programs in business, nursing, and education. 

Evening, Saturday and online courses are offered at more than 60 off-campus locations. 
Program and course materials are shipped to students and the university library operates 
an extensive off-campus support service. 

This program is committed to meeting the educational needs of working 
professional adults. These educational needs include the development of workplace skills 
that result in resilience, adaptability, and self-reliance. These competencies, critical for 
success in today's work environment, APS has called meta-skills - cognitive and affective 
strategies that can be successfully applied to novel situations. The focus of the current 
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study is on the impact of instructional strategies on the development of these meta-skills 
in the adult learner. 



Study Context and Purpose 

The context of this study was the Graduate Studies in Education Department of 
the Adult and Professional Studies Division of Indiana Wesleyan University. This 
graduate education program offered day-long Saturday classes for cohorts of practicing 
teachers at off-campus sites around the state. Students progressed through an integrated 
eighteen-month program and were supported by an extensive system of off-campus 
services. 

This program was committed to meeting the educational needs of working 
professionals. Responding to workplace demands for self-reliant thinkers, able to adapt 
to and thrive in changing circumstances, a focus of this graduate program was on meta- 
skill development Referred to as the "Ten Across the Curriculum" skills (see Appendix 
B), these meta-skills were cognitive and affective processes that transcended knowledge 
acquisition (American Heritage Dictionary, 1990). They were strategies that an 
individual could successfully apply to the processing of new information in an unfamiliar 
situation (Clark, 1999). Examples of the program's target meta-skills were the ability to 
read complex materials with comprehension, the ability to communicate effectively, the 
development of information literacy, and the ability to consistently exhibit ethical thought 
and action, among others. 

Within this context, the primary purpose of this study was to determine whether 
the instructional strategies imbedded in the program's design were related to the 
development of adult learners' meta-skills. For example, were class activities in the 
course modules related to the attainment of students' meta-skills? Were assessment 
procedures related to meta-skill development? Was meta-skill growth related to high 
expectations, instructors' worldview, or effective use of textbooks? 
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Theoretical Framework 



The theoretical framework of this study was the social learning theories of 
Johnson and Johnson (1994) and Rotter (1954). The Social Interdependence Theory of 
Johnson and Johnson assumed the way social interdependence was structured determined 
how individuals interact, which, in turn, determined behaviors. Positive interdependence 
resulted in cooperative interaction through which individuals encouraged and facilitated 
each other's efforts. Negative interdependence resulted in competitive interaction, as 
individuals discouraged and obstructed each other's efforts to achieve. 

In this study context, social interdependence was structured through the non- 
traditional format of the graduate education program. Course activities and assignments 
were designed to develop positive social interdependence and encourage collaborative 
interaction. Instructional strategies within this format were dependent upon students' 
willingness to engage in the learning context, and instructor's ability to facilitate positive 
social interdependence. Students were expected to consolidate personal experience and 
expertise with the shared understandings of the cohort and to gain content knowledge and 
meta-skills through the interaction. 

The Social Learning Theory of Julian Rotter arose from the traditions of both 
learning theory and personality theory. Rotter established the assumption that 
meaningful learning occurred in a social environment, through social interactions with 
other people. "It is a social learning theory because it stresses the fact that the major or 
basic modes of behaving are learned in social situations and are inextricably fused with 
needs requiring for their satisfaction the mediation of other persons" (Rotter, 1954, p. 84). 

A theoretical assumption of this study was that positive social interdependence 
and collective affective and cognitive interactions stimulate meaningful learning. The 
program's structure of study groups and cohort learning communities supported learners' 
acquisition of knowledge and provided a framework that fostered learner meta-skill 
development. 

Another theoretical assumption of this study stemmed from the humanistic 
orientation of Carl Rogers, who, in his theory of learning, stated that the learner was best 
situated to evaluate his or her learning experience and was able to assess cognitive and 
affective growth (1983). Self-reports by students on end of course surveys constituted 
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the data collected in the current study. These self-reports described students' learning 
experiences and their perceptions of meta-skill growth. Based on Rogers' learning 
theory, it was assumed that participants in this study were self-aware individuals, able 
and willing to evaluate accurately and honestly their learning experiences. 

Review of Literature 

Instructional effectiveness is a critical concern for stakeholders across the 
spectrum of educational contexts and constituencies. From preschool classes to GED 
courses, in charter schools and colleges, educators strive to design optimal learning 
opportunities. The research literature reflects the concern and desire of educators to 
implement instructional strategies that maximize student learning. 

For the purposes of this study, the literature base on instructional effectiveness 
has been narrowed to the field of higher education. It has been further narrowed by 
focusing only on those studies that have concentrated on the impact of instructional 
strategies on student learning. The two areas of research reviewed are the following: 

> studies conducted to clarify the relationship of instructional strategies on 
knowledge acquisition, and 

> studies conducted to identify instructional strategies related to meta-skill 
development. 

Studies conducted to clarify the relationship of instructional strategies to 
knowledge acquisition are plentiful, and provide a useful foundation for research on 
meta-skill development. Several excellent literature reviews of educational research have 
been published. The most comprehensive review is Pascarella and Terenzini's How 
College Affects Students (1992). Drawing upon hundreds of sources, this review 
describes findings relating to such topics as teacher behavior, students' cognitive growth, 
and instructional approaches. Less comprehensive, but more focused, is the literature 
review by Lowman (1994). A review based on studies using end-of-course survey data, 
Lowman's analysis describes what constitutes masterful teaching for content in the higher 
education environment. Focusing on a broader base, Forsyth and McMillan (1994) 
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reviewed research literature that informed educators on instructional practices effective in 
motivating students in higher education. 

Studies have also been conducted in adult education contexts. Darling-Hammond 
and McLaughlin (1995), Miller and Silvernail (1994), Kerchner (1993), Andrew and 
Schwab (1995) and Shin (1994), to name a few, have described several strategies as 
especially effective for the adult learners' knowledge acquisition. These studies in adult 
education have been set within professional development settings, as well as formal and 
informal education settings. 

Two meta-analyses have been conducted of studies focusing on the relationship 
between instructional strategies and student achievement. Cohen (1987) and Feldman 
(1994) have analyzed over 40 studies based on student ratings of specific instructor 
strategies. Studies were chosen by Cohen (1987) and Feldman (1994) that used end-of- 
course survey data, as well as end-of-course exam scores to determine both the students' 
perceptions of instructional effectiveness as well as their knowledge acquisition. Cohen 
(1987) and Feldman (1994) found, through their respective meta-analyses, the 
instructional strategies most strongly related to knowledge acquisition were the 
following: 

1 . preparation and organization of professor, 

2. clarity and understandableness of professor, 

3. sensitivity to, and concern with, class level and progress, and 

4. stimulation of student interest. 

This meta-analysis is especially informative for instructors in higher education who are 
seeking to improve student learning in their classrooms. 

Studies conducted to identify instructional strategies related to meta-skill 
development relate directly, rather than foundationally, to this study which is "centrally 
concerned with the development of a critically aware frame of mind, not with the 
assimilation of previously defined skills or bodies of knowledge" (Brookfield, 1995, p 
201). This critically aware frame of mind includes an understanding of meta-skills such 
as metacognition, the development of critical thinking and problem solving skills, and 
progress toward self-regulation and direction 
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Educators have begun to recognize the importance of teaching metacognitive 
strategies in higher education to improve the use of meta-skills and the transfer of 
learning to novel situations (Bruning, Schraw, and Ronning, 1995). Metacognitive 
awareness implies that individuals can describe their own understanding and use the 
information they have (Hammann and Stevens, 1998). 

In a study by Mandl, Gruber and Renkl (1993), strategies, such as peer 
collaboration and meta-skill application, were developed to make metacognitive teaching 
more effective. These strategies were based on two models of learning: cognitive 
apprenticeship (Collin, Brown, and Newman, 1989) and anchored instruction 
(Bransford, Sherwood, Hasselbring, Kinzer & Williams, 1990). 

Studies by Williams (1992), Glaser (1991) and Schmidt (1993) emphasized 
attention to context when teaching metacognitive strategies. Other studies (Darling- 
Hammond, 1999; OTA, 1995; Qin, Johnson and Johnson, 1995; Slavin, 1996) discussed 
the potential for metacognitive growth in collaborative settings. Meta-skill training and 
practice were highlighted in both Svinicki's (1994; 1996) and Mevarech's (1999) 
conclusions. Strategies such as scaffolding and advanced organizers have also been 
found effective (Schraw and Graham, 1997). 

The development of meta-skills such as critical thinking and self-regulation has 
been studied extensively in the past few years. Training for critical thinking is a 
challenge, as there are few models to guide educators. Strategies such as collaborative 
peer and instructor support (Keeley, Sheinberg, Cowell, and Zinnbauer, 1995), active 
learning activities (Kirby and Goodpastor, 1995), use of credible experts (Haas and 
Keeley, 1998), in-class dialogue (Jones and Ratcliff, 1993) and the use of creative 
evaluation instruments (Ramsden, 1992) have been studied and found effective in 
meeting this challenge. 

Self-regulation in learning is a complex activity, requiring an integration of 
several meta-skills (Pintrich and DeGroot, 1990). For individuals to be self-regulating, 
they must combine metacognitive processes, motivational orientation, learning strategies, 
and complex performances (Hammann and Stevens, 1998). Strategies that encourage 
self-regulation include providing accountability (Arredondo and Rucinski, 1994), 
allowing for choice and control over learning activities (Borkowski, Carr, Rellinger and 



Pressley, 1990; Pintrich, Smith, Garcia, and McKeachie, 1991) and developing 
collaborative work groups (Van Zile-Tamsen, 1997). 

Research literature on instructional effectiveness in higher education is abundant. 
Studies clarifying relationships between instructional strategies and knowledge 
acquisition for college-age and adult students provide a foundation for the current study. 
This foundation consists of literature reviews, meta-analyses and individual studies that 
focus on the relationship between instructional strategies and cognitive growth. Directly 
informing this study are several research projects that focus on the relationships between 
instructional strategies and meta-skill development. These studies explore the meta-skills 
of metacognition, critical thinking and self-regulation, and propose models and strategies 
for their development. 



Study Methodology 

Over a 12-month period, from June 1998 through May 1999, 2,567 end-of-course 
survey responses were collected from students in the Graduate Studies in Education 
program at Indiana Wesleyan University. Students completed these surveys (see 
Appendix A for survey sample) during the final class of each of eight graduate education 
courses. 

This end-of-course survey was divided into two sections. Nineteen questions 
focused on instructional strategies. Students were asked to rate the effectiveness of 
instructional strategies on a five-point Likert scale from "not at all effective (1)" to 
"completely effective (5)." 

Ten questions focused on meta-skill growth. Students were asked to rate the 
effectiveness of the completed course in helping them develop specific meta-skills. 

Again, the ratings were on a five- point Likert scale ranging from "not at all effective (1)" 
to "completely effective (5)." The surveys were anonymous, and were seen by instructors 
only after the APS Research Department had compiled the responses. 

Using the statistical program SPSS, several statistical analyses were conducted 
using the data collected from the 2,567 end-of-course surveys. These analyses tested for 
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correlations between the independent variables (instructional strategies) and each of ten 
dependent variables (meta-skills). 

First, a Pearson correlation coefficient was run of the whole sample. Next, a 
multiple regression analysis was conducted of the whole sample This analysis 
highlighted correlations between specific instructional strategies and specific meta-skills. 

A factor analysis was then conducted on the sample, using a varimax rotation. 
This analysis found four key clusters or composites of instructional strategies These 
clusters became "Composite Instructional Strategies #1 through #4." 

A multiple regression analysis was conducted of the whole sample, testing the 
correlation between the four Composite Instructional Strategies (the independent 
variables) and each of the ten meta-skills (the dependent variables). This multiple 
regression analysis was conducted again, while controlling for course. Finally, this 
multiple regression analysis was conducted while controlling for a selection of 
instructors. 

Interview evidence was also collected to provide a different perspective on the 
relationship between the program's instructional strategies and students' perceived meta- 
skill growth Focus group interviews were held with five randomly selected cohort 
communities, and included 85 respondents. 



Findings 

N: 2,567 responses 

1. MULTIPLE REGRESSION ANALYSIS OF THE WHOLE SAMPLE 



A Correlation of the 19 instructional strategies with the 10 meta-skills 



META-SKILL 


Instructor Strategy 


beta 


1. World view 


13. Faith evident 


.35 


2. Ethics 


13. Faith evident 


.21 


3. Lifelong learning 






4. Reading 


19. Textbook 


.30 


5. Critical thinking 






6. Problem solving 






7. Writing 


16. Homework 


.20 


8. Oral 






9. Finding information 


16. Homework 


.20 


10. Teamwork 


17. Discussion 


.20 



Beta coefficient with a significance of .0 1 or better. * indicates .05 or better. 
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2. FACTOR ANALYSIS 



A factor analysis with a varimax rotation was completed on the whole sample. Four key components or 
clusters of instructional strategies emerged. The “Composite Instructional Strategies” were the following 
Composite Instructional Strategy #1 : questions 1 - 7 = Andragogy 
Composite Instructional Strategy #2: questions 8 - 10, 18 = Assessment 
Composite Instructional Strategy #3: questions 15 - 17, 19 = Curriculum 
Composite Instructional Strategy #4: question 13 = Faith 



End of 
Course 
Survey 
Question 


Composite Instructional Strategy 


# 1 : 

Andragogy 


# 2 : 

Assessment 


# 3 : 

Curriculum 


# 4 : 

Faith 


1 


.68 








2 


.61 








3 


.70 








4 


.72 








5 


.72 








6 


.50 








7 


.57 








8 




.64 






9 




.84 






10 




.84 






11 










12 










13 








69 


14 










15 






.69 




16 






.76 




17 






.68 




18 




.66 






19 






.71 





3. MULTIPLE REGRESSION ANALYSIS OF THE FOUR COMPOSITE 
INSTRUCTIONAL STRATEGIES WITH THE TEN META-SKILLS 



META-SKILL 


Andragogy 


Assessment 


Curriculum 


Faith 


1. World view 






.27 


.34 


2. Ethics 


.21 




.33 


.21 


3. Lifelong 


.27 




.35 




4. Reading 






.50 




5. Critical thinking 


.24 




.40 




6. Problem solving 






.42 




7. Writing 






.37 




8. Oral 


.21 




.36 




9. Information 






.43 




10. Teamwork 






.30 





Beta coefficient with a significance of .01 or better. * indicates .05 or better. 
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4. MULTIPLE REGRESSION ANALYSIS OF THE FOUR COMPOSITE 
INSTRUCTIONAL STRATEGIES WITH THE TEN META-SKILLS 
CONTROLLING FOR COURSE. 

QgmgosUe Instructional Strategy # 7 : Andragogy 



METASKILL 


Edu545 

Educ. Issues 


Edu550 

Curriculum 


Edu551 

I nstr. Theory 


Edu553 

Assessment 


Edu554 

Computers 


Edu555 

Materials 


Edu556 

Research 


Edu557 

Leadership 


1 . World view 












.32 






2. Ethics 










.27 




.26 


.37 


3. Lifelong 




.32 


.20 




.32 


.32 


.33 


.29 


4. Reading 


















5. Critical thinking 






.25 




.25 




.24 




6. Problem solving 










.28 






.36 


7. Writing 


.28 








.27 








8. Oral 


.22 


.24 




.31 


.26 


.24 






9. Information 


















10. Teamwork 



















Beta coefficient with a significance of .01 or better. * indicates .05 or better. 



Composite Strategy #2: Assessment 



METASKILL 


Edu545 

Educ. Issues 


Edu550 

Curriculum 


Edu551 

Instr. Theory 


Edu553 

Assessment 


Edu554 

Computers 


Edu555 

Materials 


Edu556 

Research 


Edu557 

Leadership 


1 World view 


















2. Ethics 






.21 












3. Lifelong 


















4. Reading 


















5. Critical thinking 


















6. Problem solving 


















7. Writing 




.24 


.26 












8. Oral 






.30 








.30 




9. Information 






.21 












10. Teamwork 










.22 


.31 


.29 





Beta coefficient with a significance of .01 or better. * indicates .05 or better. 



Compos/te Instructional Strategy #3; Curriculum 



METASKILL 


Edu545 

Educ. Issues 


Edu550 

Curriculum 


Edu551 

Instr.Theory 


Edu553 

Assessment 


Edu554 

Computers 


Edu555 

Materials 


Edu556 

Research 


Edu557 

Leadership 


1 . World view 


.27 




.46 


.28 


.25 


.34 




.24 


2. Ethics 


45 


.32 


.35 


.40 


.25 


.39 


.20 


.26 


3. Lifelong 


4 


.29 


43 


.41 


.28 


.33 




.31 


4. Reading 


.49 


.44 


.65 


.47 


.35 


.37 


.58 


.51 


5. Critical thinking 


.37 


.40 


.46 


.44 


.31 


.37 


.43 


.34 


6. Problem solving 


. .37 


.50 


.50 


.45 


.29 


.36 


.58 


.28 


7. Writing 


.37 


.36 


.43 


.38 


.26 


.43 


.48 


.22 


8. Oral 


.40 


.40 


.41 


.35 


.26 


.37 


.38 


.24 


9. Information 


.46 


.46 


.50 


.40 


.37 


.30 


.51 


.20* 


10. Teamwork 


.31 


.24 


.33 


42 


.31 









Beta coefficient with a significance of .01 or better. * indicates .05 or better. 
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Composite Instructional Strategy #4. Faith 



METASKILL 


Edu545 


Edu550 


Edu551 


Edu553 


Edu554 


Edu555 


Edu556 


Edu557 




Educ. Issues 


Curriculum 


I nslr. Theory 


Assessment 


Computers 


Materials 


Research 


Leadership 


1 . World view 


.22 


.30 


.25 


.45 


.40 




.32 


.40 


2. Ethics 




.20 




.23 


.26 




.20 




3. Lifelong 














.20 




4. Reading 
















.23 


5. Critical thinking 
















.20 


6. Problem solving 


















7. Writing 
















.30 


8. Oral 
















.35 ! 


9. Information 


















10 Teamwork 



















Beta coefficient with a significance of .0 1 or better. * indicates .05 or better. 



5. MULTIPLE REGRESSION ANALYSIS THE FOUR COMPOSITE 
INSTRUCTIONAL STRATEGIES WITH THE TEN META-SKILLS 
CONTROLLING FOR INSTRUCTOR. 

The sample of instructors consists of all instructors with 60 or more end of course survey responses in the 
data bank. Eight were chosen out of 73 instructors. 



Composite Instructional Strategy #L Andragogy 



METASKILL 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 




1 


2 


3 


4 


5 


6 


7 


8 


1 . World view 








.46 


.33 








2. Ethics 


















3. Lifelong 










.31 




.33 




4. Reading 


















5. Critical thinking 






.41 


.40 










6. Problem solving 








.26* 


.32 


.38* 






7. Writing 


















8 Oral 














.36* 




9. Information 






.34 












10. Teamwork 






.37 


.58 


.24* 


.40* 







Beta coefficient with a significance of .0 1 or better. * indicates .05 or better. 



Composite Instructional Strategy # 2 : Assessment 



METASKILL 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 




1 


2 


3 


4 


5 


6 


7 


8 


1 . World view 


















2. Ethics 


















3. Lifelong 














-.48 




4. Reading 


.28* 












-.50 




5. Critical thinking 


.38 












-.58 




6. Problem solving 


.54 


.41* 










-.40* . 




7. Writing 
















.50 


8 Oral 


















9. Information 


.36 






.23* 


.33 




-.48 




10 Teamwork 


.39 r 














.43* 



Beta coefficient with a significance of .01 or better. * indicates .05 or better. 
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Compos/Ye Instructional Strategy #3: Curriculum 



METASKILL 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 




1 


2 


3 


4 


5 


6 


7 


8 


1. Worldview 


.36 


.48 


.48 


.41 


.45 




.60 


.38* 


2. Ethics 


.53 


.53 


.57 


.30* 


.28* 


.34* 


.60 


.50 1 


3. Lifelong 


.45 


.49 


.34 


.35 


.45 


.38* 


.70 


.52 


4. Reading 


.38 


.55 


.47 


.63 


.55 


.53 


.88 


.42 


5. Critical thinking 


.38 


.60 


.36 


.48 


.45 


.50 


.77 


.50 


6. Problem solving 


.30 


.37 


.45 


.54 


.31 


.50 


.67 


.52 


7. Writing 




.63 


.34 


.48 


.49 


.50 


.53 


.34 


8. Oral 


.33 


.54 


.46 


.51 


.35 


.64 


.44 


.36 


9. Information 


.50 


.58 


.37 


.60 


.40 


.51 


.75 


.42 


10. Teamwork 




.64 


.30 


.27 


.31 


.80 


.43 





Beta coefficient with a significance of ,01 or better. * indicates .05 or better. 



Composite Instnictiongj Strategy #4: Faith 



METASKILL 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 


Instructor 




1 


2 


3 


4 


5 


6 


7 


8 


1 . World view 






.38 




-.24 


.31* 


.52 




2. Ethics 






.20 








.33* 




3. Lifelong 




.29 






-.25 




.28 




4. Reading 














.48 




5. Critical thinking 




.37 






-.23* 




.46 




6. Problem solving 














.54 




7. Writing 










-.23* 








8. Oral 




.33 














9. Information 




.34 










.24* 




10. Teamwork 



















Beta coefficient with a significance of .01 or better. * indicates .05 or better. 
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Discussion 



The purpose of this study was to determine whether instructional strategies 
imbedded in the program design of the study context were related to the attainment of 
adult learners' meta-skills. Instructional strategies were represented in this study by 19 
end-of-course survey questions and were defined as characteristics and techniques 
implemented in the instructional context to promote learning. Meta-skills were 
represented by 10 "Across the Curriculum" skills and were defined as cognitive and 
social processes that transcend knowledge acquisition and that are used to process new 
information in novel situations (Clark, 1999; Jacobson, 1998). 

Three primary findings resulted from this study. First, clusters of end-of-course 
survey questions, hereafter referred to as instructional strategies, tended to function 
together in relation to students' reported meta-skill attainment. Four such clusters, or 
composite instructional strategies, were identified. Second, the c urriculum composite of 
instructional strategies strongly correlated with reported attainment of all ten meta-skills. 
Third, the a ssessment composite of instructional strategies correlated with reported 
attainment of none of the meta-skills. 



The first statistical test run on the data collected from the 2,567 end-of-course 
surveys was a Pearson correlation coefficient [r]. Analysis of the Pearson's r raised 
concerns of multi-colinearity. Many instructional strategies seemed to be interrelated. 
Multi-colinearity tends to give false positive results with statistical tests such as Pearson's 
r and multiple regression analysis. An analysis of the results from the Pearson's r led to a 
factor analysis of the whole sample. 

Before the factor analysis was performed, a multiple regression analysis of the 
whole sample was completed. This analysis was a correlation of the 19 instructional 
strategies as independent variables with the ten meta-skills as dependent variables. This 
whole sample multiple regression analysis demonstrated that specific meta-skills (the 
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dependent variables) strongly correlated with specific instructional strategies (the 
independent variables). Four instructional strategies significantly related to the reported 
attainment of seven meta-skills. Three meta-skills were not significantly related to 
specific instructional strategies (see Findings, Chart 1 , p. 8) 

Exploring the relationship between individual instructional strategies and reported 
meta-skill attainment was the initial purpose of this study. However, upon running a 
factor analysis on the data, it was found that these individual strategies clustered into 
identifiable composites. The instructional composites provided a clearer, more 
significant picture of the relationship between instructional strategies and reported meta- 
skill attainment 

Instructional Composites 

The factor analysis identified four composites of instructional strategies that were 
related to the reported attainment of specific meta-skills in similar ways. These four 
composites were named according to their defining characteristics: 

> Andragogy, Composite Instructional Strategy #1, clustered end-of-course 
survey questions 1 through 7: 

1 The instructor's professional and academic experiences were 
appropriate to teach this course 

2. The instructor made good use of the allotted class time 

3. The instructor was effective in helping students share their 
professional knowledge 

4. The instructor treated me and my colleagues as professional adults 

5. The instructor was effective in encouraging and motivating me and my 
colleagues 

6. The instructor taught the learning objectives set out in the curriculum 
materials 

7. The instructor made it clear why it was important to learn the subject 
matter 

> Assessment . Composite Instructional Strategy #2, clustered end-of-course 
survey questions 8 through 10, and 18: 
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8. The instructor established clear criteria for grading students' 
performance 

9. The instructor's grades accurately reflected students' performance 

10. The instructor provided detailed feedback on the quality of graded 
assignments 

18. The grading process measured what I actually learned, performed, and 
produced. 

> Curriculum . Composite Instructional Strategy #3, clustered end-of-course 
survey questions 15 through 17, and 19. 

15. The module/syllabus is clearly written 

16. The outside activities/assignments helped me to understand and 
make use of the knowledge I gained from this course 

17. The in-class activities/assignments helped me to discuss key 
concepts and experiences with other adult learners 

19. I was able to understand the information in the textbook 

> Faith . Composite Instructional Strategy #4, was end-of-course survey question 
#13: 

13. The instructor's faith was evident during the course 

(For Factor Analysis results please see Findings, Chart 2, p 9) 

A primary finding of this study was that clusters of instructional strategies tended 
to function together in relation to students' reported meta-skill attainment. Although 
these composite instructional strategies were identified through a factor analysis, each 
composite was supported by a recognized educational theory. The theoretical support for 
each composite hypothesized the positive relationship between specific strategies and 
student growth, explained how individual strategies complemented one another, and 
described the relationships between these instructional strategies. 

Andragogv . Composite Instructional Strategy #1, was drawn from seven 
strategies (end-of-course survey questions 1 through 7) that are descriptors of Knowles' 
"theory of adult learning" (Pratt, 1993, p. 16) Among these strategies were the 
following. 
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^ incorporating students' professional experiences into the learning context; 

^ treating students as professional adults; 

> making clear the relevance of course content to student's professional lives; 
and 

> using a variety of resources effectively. 

This non-traditional program was intentionally designed to meet the needs of adult 
learners. This finding confirmed the positive relationship between the strategies of 
andragogy and students' reported attainment of meta-skills. 

Assessment . Composite Instructional Strategy #2, was drawn from four strategies 
(end-of-course survey questions 8, 9, 10 and 18) that described, in part, current 
assessment theory's construct of effective assessment. Assessment strategies that 
provided clear criteria, that accurately reflected performance, and that were timely and 
comprehensive composed this composite and are included in this theoretical construct 
(Wiggins and McTighe, 1998). Current assessment theory also claims that assessment 
fosters learning (Porter, 2000). This claim was not supported by this study, for the 
assessment composite was found to be unrelated to students' reported attainment of meta- 
skills. 

Curriculum , Composite Instructional Strategy #3, was drawn from four strategies 
(end-of-course survey questions 15, 16, 17, and 19) that clearly supported the social 
learning theories (Johnson & Johnson, 1994; Rotter, 1954) upon which this study and the 
program were based. The curriculum composite consisted of the curricular "how" (in- 
class and out-of-class activities and assignments) and the "what" (the course module and 
textbooks) of the learning environment. Social learning theory claims that interactive and 
interpersonal course activities and assignments, coupled with substantive content, result 
in student learning This instructional composite confirmed that collaborative learning 
strategies and substantive content aggregated and together were positively related to 
students' reported attainment of meta- skills. 

Faith , Composite Instructional Strategy #4, was drawn from one strategy (end-of- 
course survey question 13), "The instructor's Christian faith was evident." This strategy 
supported Jourard's Interpersonal Communication Theory (1971) which states that self- 
disclosure builds trust. Trust, in turn, creates a safe learning environment in which 
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students can explore challenging, often personal, issues Ostermann (1990) stated that 
self-disclosing instructors, who make personal values public, are more likely to stimulate 
students to explore their values. This finding supported Jourard's theory, as instructors' 
self-disclosure of their faith was positively related to students' reported attainment of 
meta-skills 1 and 2: the ability to see things from the perspective of a Christian 
worldview and skill in ethical thought. 

It should be mentioned that three end-of course survey questions were not 
significantly related to any instructional strategy: 

> The instructor set high standards for achievement in this course (end-of-course 
question #11) 

> The instructor was accessible to me beyond the regular class period (end-of- 
course question #12) 

> On the average, our class began and ended on time (end-of-course question 
#14) 

Given the nine hour block of instructional and relational time for each class period in this 
non-traditional program, it was not surprising that beginning and ending class on time 
and instructor accessibility were not important issues to students. 

It was surprising that the level of expectations set by the instructor seemed 
unrelated to reported meta-skill attainment. A plethora of educational research supports 
the importance of setting high expectations to increase student learning. The discrepancy 
between this study's findings and the predominance of research literature highlights an 
area in need of further investigation. 



The factor analysis consolidated the original 19 instructional strategies into four 
composite instructional strategies. A multiple regression analysis was then run to test the 
correlation of the four composite instructional strategies (the independent variables) with 
the reported attainment of each of the ten meta-skills (the dependent variables) (see 
Findings, Chart 3, p. 9). 

To validate that these correlations were independent from the courses taught and 
the instructors who taught them, multiple regression analyses were then run while 




20 



17 



controlling for course and selected instructors (see Findings, Charts 4 and 5, pp 10-12) 
When controlling for course and instructor, the same correlations of consolidated 
instructional strategies with reported meta-skill attainment occurred as with the whole 
sample: 

> Andragogy correlated moderately (.21- 27) with meta-skills 2, 3, 5, & 8 

> Assessment correlated insignificantly across all meta-skills 

> Curriculum correlated strongly (.27- 50) across all meta-skills 

> Faith correlated moderately (.21- 24) with meta-skills 1 & 2 

Curriculum Composite 

A second primary finding of this study was that the curriculum composite of 
instructional strategies strongly correlated with reported attainment of all ten meta-skills. 
This curriculum composite consisted of four instructional strategies. Two strategies 
focused on context: survey questions #16 and #17 stated that the in-class and out-of-class 
activities and assignments fostered learning and application of class content. Two 
strategies focused on content: survey questions #15 and #19 stated that the module was 
clear and the textbook was understandable. 

Clear and understandable content, taught in a student-centered, collaborative 
context, resulted in the strongest correlations with reported meta-skill attainment of the 
four composite instructional strategies. The educational design of this program was 
based on a student-centered, cognitive model of instruction. That is, the goal of the 
teaching/learning process was the intellectual and professional maturation of the student. 
Content and instructional strategies were chosen to foster the cognitive development of 
the student, who was viewed as a self-reliant, self-directing individual. 

Teaching and learning always occur in some context. The context of this study 
was based on social learning theory and designed to be student-centered and 
collaborative. A large body of research confirmed the findings of this study that a 
student-centered, collaborative context supports meta-skill attainment. 

Johnson, Johnson, and Smith (1994) found that greatest cognitive growth 
occurred when students worked collaboratively and were rewarded as a group for 
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academic performance Ahem-Rindell (1998), in a recent case study, concluded that the 
use of cooperative work groups and inquiry-based activities contributed importantly to 
students' meta-skill development. Smith (1994) found that active participation and peer- 
to-peer interaction consistently emerged as being significant and positively related to 
students' meta-skill growth, especially critical thinking. 

In focus group interviews, students reported that small group work and 
application of course content to their work environment facilitated their growth in meta- 
skills. One student commented, "Trusting my cohort group, working together toward a 
common goal, completing my course assignments in my workplace - these are the things 
that made me grow. Especially my ability to work in a team, and in problem solving and 
critical thinking skills." Another student remarked, "Being held accountable by a group 
whose respect I wanted, studying and completing assignments together, working with my 
instructors and my supervisors - all these forced me to learn how to solve problems, how 
to communicate, how to find materials I needed, how to apply my knowledge in my 
fourth grade classroom." 

Wilson (1993) advocated for locating learning within the actual situation of its 
creation and use. In this educational environment, learning was placed in its natural 
context. Activities and assignments were focused upon students' professional experience 
and environment This study's findings supported the conclusion that this curricular 
context facilitated students' integration of knowledge and cultivation of all ten meta- 
skills. 



The curriculum composite also consisted of instructional strategies that focused 
on clear and understandable content. In conjunction with the student-centered, 
collaborative context, this content resulted in the strong correlations with reported meta- 
skill attainment. Because the goal of this educational program was the intellectual and 
professional maturation of the student, content was chosen to foster students' cognitive 
development. 

Several studies supported the contention that "a robust content," coupled with 
interactive instructional strategies, result in meta-skill attainment (Palincsar, 1989) 
Pintrich and Garcia (1994) concluded that meta-skill attainment was a cyclical process in 




19 

22 



which meta-skill development influenced goal orientation, which in turn affected 
cognitive engagement with content, which resulted in increased performance levels, 
which led to further meta-skill development. Garner (1990) concluded that effective 
meta-skill development resulted from learning vigorous content and implementing a 
variety of cognitive strategies. Billson and Tiberius (1994) found evidence that effective 
teaching and learning was dependent upon the breadth and depth of content, in 
conjunction with active inquiry in a supportive atmosphere. Dressel and Marcus (1994) 
concluded that content chosen to be interesting, stimulating, and productive, when 
combined with teaching methods that encourage students' engagement, fostered high 
levels of meta-skill development 

In this study's educational context, content was selected for its currency, its 
applicability and relevancy to the student's workplace, and for its contribution for the 
student's knowledge base. When combined with active and interactive instructional 
strategies, students reported this program's content was positively related to their meta- 
skill attainment across all ten meta-skills. In a focus group interview, one student 
commented, "The content was challenging. It stretched me, but not to the breaking point. 
It forced me to think, to process, to reflect on what I was doing." Another student stated, 
"The textbooks were dry the first read-through. But then, when we discussed and applied 
the information, it came alive. The application of course content - that was what did it 
for me. I could use the material then." 

Students reported attainment of all ten meta-skills as a result of the curriculum 
composite of instructional strategies. This finding substantiated the large body of 
educational research that has concluded that a strong content and an interactive context 
result in student growth. The relationship between the curriculum composite and 
reported meta-skill attainment was clearly the strongest relationship found in this study. 

Assessment Composite 

A third primary finding of this study was that the assessment composite of 
instructional strategies correlated with attainment of none of the meta-skills. 

The assessment composite consisted of four instructional strategies: clear grading criteria, 
timely and adequate feedback, and accurate reflection of performance Through factor 
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analysis it was found that these four instructional strategies tended to function together in 
relation to students' reported meta-skill attainment. Through the multiple regression 
analysis it was found that there were no statistically significant correlations between the 
assessment composite and students' reported meta-skill attainment. 

Wiggins and McTighe (1998) have stated the effectiveness of assessment is 
ultimately determined by its achievement of desired learnings. Danielson claims it is 
through assessment that students advance their understandings (1996). In this study, 
assessment was unrelated to meta-skill attainment. Therefore, according to current 
assessment theory, students in this program were potentially inhibited from achieving 
desired learnings and advancing their understandings. 

In focus group interviews, several student comments were directed to this 
assessment issue. One student observed, "This program is schizophrenic. On the one 
hand, we apply, apply, apply our new knowledge to our workplaces. On the other hand, 
we are tested, tested, tested on what we learn The application should be the test! We 
should be asked - Can you apply your new knowledge successfully? Prove it!" Another 
student commented, "We're adult learners. Why are we being given midterms and final 
exams? Why are we writing long research papers? Our weekly assignments are based in 
our elementary classrooms. Our class presentations are transferable to our work. So tell 
me - why the tests? Why the papers?" Judging from interview responses, assessment 
practices were resented and seen as obstructions to learning. 

The most current thinking among educators is that assessment reform is needed 
across educational contexts (Bond, 2000). Bond, Herman and Arter (1994) have stated 
that we must change our assessment strategies to tie assessment design and content to 
new outcomes and purposes for assessment. A new purpose for assessment is to foster 
learning of worthwhile academic content, while providing educators with evidence for 
making decisions about curriculum and teaching (Porter, 2000). The new assessment 
design and content has been identified as authentic and performance-based, and is 
regarded as more reflective of new curricular goals and methods of instruction (Geltner, 
1993; National Commission on Teaching and America's Future, 1996). 

In this study context, assessment practices were primarily traditional and 
consisted of exams, research papers, article critiques, and presentations. Program 
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designers have been incorporating more authentic, performance-based assessments into 
the program's courses as a response to student input and current assessment theory. 
Clearly, the evidence from this student sample supported this move from traditional 
assessment practices to assessments demonstrating application of course content. 

Students reported no attainment of meta-skills as a result of the assessment 
composite of instructional strategies. The relationship between this assessment 
composite and reported meta-skill attainment was the weakest relationship found in this 
study. This evidence can be viewed as dis-confirming of current assessment theory, 
which states that assessment leads to learning. It can more reasonably be viewed as 
confirming current assessment theory and highlighting a need for more authentic, 
performance-based assessment practices in the program under study. 

Implications for Practice 

Three key implications were drawn from this study: the value of a strong 
curriculum, the importance of appropriate assessment procedures, and the merit of 
intentionally developing learners' meta-skills. 

The importance of a strong curriculum, both in context and content, was 
highlighted in this study. It was found that substantive curriculum was strongly related to 
students' reported attainment of meta-skills across all categories. This finding implied 
that resources spent on curriculum development were a worthwhile investment for this 
educational institution, whose goal was to facilitate student growth. 

The strong relationship found between curriculum and student learning also 
implied that Johnson and Johnson's Social Interdependence Theory was applicable to the 
adult learning context. This social learning theory has been studied in elementary and 
secondary as well as traditional higher education settings Little or no research has been 
conducted testing this theory in an adult education context. This study's findings support 
the use of Johnson and Johnson's Social Interdependence Theory in a non-traditional, 
higher education setting 

The importance of appropriate assessment procedures was also highlighted in this 
study. It was found that assessment practices were unrelated to students' reported meta- 



skill attainment. Clearly, assessment practices in this study's context must be redesigned 
to support student learning and contribute to the development of meta-skills. 

The success of teaching should be determined by whether students learn, not by 
what the instructor does. Assessment strategies must provide evaluation opportunities 
that offer a picture of what has been learned, while supporting the learning process. For 
the adult student, assessment must be problem-centered and application-based to 
facilitate the development of meta-skills. 

The importance of intentionally developing learners' meta-skills was another 
implication of this study. As knowledge bases expand, meta-skills such as critical 
thinking, problem solving, and decision-making become critical for the intellectual and 
professional survival of adult learners. Pintrich and Garcia (1994) concluded that meta- 
skill development led to increased cognitive engagement and learning, which in turn led 
to meta-skill development. If learning is rooted in meta-skill development, then overtly 
training students in the use of meta-skills should enhance the learning process. 

Tishman, Perkins and Jay (1995) described a model for teaching students meta- 
skills. They stated that these skills, when taught intentionally, "became habits which led 
to a future of effective problem solving, thoughtful decision making, and life-long 
learning" (p. 1). Surely students' attainment and reinforcement of these habits are the goal 
of every adult educator. 

Implications drawn from this study were the following. 

> Curriculum and assessment are essential components of any educational 
program and require substantial resource investment and attention to design. 

> Direct meta-skill instruction should be a component of courses offered in 
formal adult education programs. 

Recommendations for Further Research 

In some respects, the current study raised more questions than it answered. 
Therefore, these research recommendations were focused narrowly on suggestions for 
studies of the relationship between instructional strategies, instructor characteristics, and 
student learning. A cautious approach to study design should contribute to a more 
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scholarly understanding of this relationship and researchers must keep in mind that they 
will find no simple formula for good teaching 

The following relationships and concepts deserve further study: 

■ What was the relationship between curriculum strategies of context and 

content? 

■ What was the relative influence of each? 

■ How did they impact each other? 

■ Were they equally effective as separate strategies, or was the combination 
synergistic? 

■ What influence did individual courses have upon the effectiveness of the 
curriculum for enhancing learning? 

■ What was the relationship among assessment strategies? 

■ Was the effectiveness of certain strategies canceled by the ineffectiveness 
of others? 

■ What influence did individual instructors have upon the effectiveness of 
assessment strategies to enhance learning? 

■ In what ways did individual course goals and assignments influence the 
relationship between assessment strategies and learning? 

■ In this context, what was the relationship between the instructional strategy of 

setting high standards and meta-skill development? 

■ What was the relationship between individual instructors' expectations and 
student learning? 

■ What was the relationship between individual course goals and 
assignments, instructor's expectations, and student learning? 

■ In what ways did cohort demographics (geographic location, gender 
distribution, age, cohort size) influence expectations and student learning? 

■ What was the relationship between individual faculty member's strategies and 

goals and student learning? 

■ In what ways did individual instructors implement the general strategies 
imbedded in the program's design? 

■ How did instructional variations influence student learning? 

■ How did differing goals influence student learning? 

■ What characteristics of the curricula were related to meta-skill attainment? 

■ Did the program's courses tend to work together to enhance student 
learning? 

■ Did conflicts among course goals, processes and outcomes reduce student 
learning? 

■ What course characteristics were more strongly related to student 
learning? 
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Conclusion 



Brookfield (2000) has stated that the research agenda in the field of adult 
education should focus on ways in which meta-skills, and in particular critical reflection, 
are learned and taught across adult educational contexts. This study focused on the 
relationship between instructional strategies and students' reported meta-skill attainment 
in an adult education context. 

The three key findings of this study were the following: 

> Instructional strategies clustered into four identifiable composites, each of 
which was supported by a current and credible educational theory; 

> The curriculum composite of instructional strategies strongly correlated with 
reported attainment of all ten meta-skills; 

> The assessment composite of instructional strategies correlated with reported 
attainment of no meta-skills. 

It is hoped the findings of this study will encourage reflectiveness about teaching 
and learning among faculty and students in and beyond this study context. As educators 
of adults, we must be asking - what can research tell us about program designs offered to 
adult learners? How can we best equip our students to compete in today's world? 

We live in an age of information. The knowledge base of many professions is 
growing exponentially. The glut of information overwhelms us and forces us to ask - 
What do our adult students really need to know and do? 

An important goal of responsible adult education in the 21 st century must be the 
development of learners' meta-skills - those skills required for productivity in 
professional jobs of the Information Age. Employers from every profession are asking 
educational institutions to equip a workforce with the necessary skills to think critically, 
to solve problems, to communicate effectively and to work productively in teams. 

Employers are requesting that learners be trained in cognitive and social processes 
that transcend knowledge acquisition. They are expecting, even insisting, that educators 
guide learners in their attainment of meta-skills. Intentionally developing educational 
institutions and instructional strategies that facilitate meta-skill growth can importantly 
impact our learners' success in the new century. 
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APPENDIX A 



End of Course Survey 



Instructions: Answer each question as precisely as you can. Circle only one response for each question. If the question does not apply to 
you leave it blank. Give the Survey to the Class Representative who will forward than collectively to the Research Specialist 



Core Group Course# Instructor 

Beginning Date Ending Date 



For proper processing please fin In all above information. 


not at 


very 


some- 


mostly 


com- 






aO 


little 


what 


pletely 


Instructor 








1. 


The instructor’s professional and academic experiences were 
appropriate to teach and facilitate this course. 


1 


2 


3 


4 


5 


2. 


The instructor made good use of the allotted class time. 


1 


2 


3 


4 


5 


3. 


The instructor was effective in helping me share my professional 
knowledge. 


1 


2 


3 


4 


5 


4. 


The instructor treated me and my colleagues as professional adults. 


1 


2 


3 


4 


5 


5. 


The instructor was effective in encouraging and motivating me and 
my colleagues. 


1 


2 


3 


4 


5 


6. 


The instructor made effective use of the textbooks and the module. 


1 


2 


3 


4 


5 


7. 


The instructor made it clear how the subject matter could enhance my 
professional growth. . 


1 


2 


3 


4 


5 


8. 


The instructor established clear criteria for grading my assignments. 
(Assignments include in-class and out-of-class work that is graded.) 


1 


2 


3 


4 


5 


9. 


The instructor’s assessment of my assignments, to date, accurately 
reflect my performance. 


1 


2 


3 


4 


5 


10. 


The instructor provided timely, adequate feedback on the quality of 
graded assignments. 


1 


2 


3 


4 


5 


11. 


The instructor set high standards for achievement in this course. 


1 


2 


3 


4 


5 


12. 


The instructor was accessible to me beyond the regular class period. 


1 


2 


3 


4 


5 


13. 


The instructor’s Christian faith was evident. 


1 


2 


3 


4 


5 


14. 


On the average, our class began and ended on time. 


1 


2 


3 


4 


5 


Curriculum 


1. 


The module/syllabus was written clearly. 


1 


2 




4 


5 


2. 


The outside activities/assignments helped me to understand and make 
use of the knowledge I gained from this course. 


1 


2 


3 


4 


5 


3. 


The in class activities/assignments helped me to discuss key concepts 
and experiences with other adult learners. 


1 


2 


3 


4 


5 


4. 


The grading procedures measured what I actually learned, performed, 
and produced. 


1 


2 


3 


4 


5 


5. 


I was able to understand the information in the textbook. 


1 


2 


3 


4 


5 



(over) 





not at 
mQ 


vary 

little 


some- 

what 


mostly 


com- 

pletely 


Ten Across Objectives 

Rate the effectiveness of this course in helping me attain: 








1 . The ability to see things from the perspective of a Christian world 
view. 


1 


2 


3 


4 


5 


2. Skill in ethical thought and action. 


1 


2 


3 


4 


5 


3. Values and skills necessary for lifelong learning. 


1 


2 


3 


4 


5 


4. The ability to read complex materials with comprehension. 


1 


2 


3 


4 


5 


5. Skill in thinking critically concerning ideas and performance (i.e. 
reflective practice). 


1 


2 


3 


4 


5 


6. Skill in problem solving and decision making 


1 


2 


3 


4 


5 


7. The ability to communicate effectively through writing. 


1 


2 


3 


4 


5 


8. The ability to communicate effectively orally. 


1 


2 


3 


4 


5 


9. The ability to find needed information (sometimes called information 
literacy). 


1 


2 


3 


4 


5 


10. The ability to work effectively in teams. 


i 


2 


3 


4 


5 



Comments: (please print legibly) 
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APPENDIX B 

Ten Across the Curriculum: Key Process Skills 

Supporting the Teacher As Agent of Change conceptual model for the Master of Education 
Degree, and undergirding the learning experience for practicing classroom teachers are the “Ten 
Across the Curriculum” learning process skills. Candidates in the M.Ed. program will practice 
these skills throughout their graduate learning experience. The “Ten Across” skills are: 

1. The ability to see things from the perspective of a Christian worldview. 

2. The ability to consistently exhibit ethical thought and action. 

3. The ability to abide by values necessary for lifelong learning. 

4. The ability to read complex materials with comprehension. 

5. The ability to think critically concerning ideas and performance (i.e. reflective practice). 

6. The ability to engage in problem solving and effective decision-making. 

7. The ability to communicate effectively through writing. 

8. The ability to communicate effectively in team meetings and presentations. 

9. The ability to find needed information (sometimes called information literacy). 

10. The ability to work effectively in teams. 

The Ten Across Skills are practiced and perfected in every M.Ed. course. These skills are included as part 
of the End of Course Survey data collected at the close of each core course. 
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